PC Open 92/PC Open 92 CD 1.bin/internet/AmphetaDesk/docs/TODO

Text File | 2002-10-29 | 21.0 KB | 389 lines

###############################################################################
# AmphetaDesk                                           (c) 2000-2002 Disobey #
# TODO                                    http://www.disobey.com/amphetadesk/ #
###############################################################################

------------------------------------------------------------------------------
SLATED FOR v0.94 RELEASE (ie. try to get these done before everything else)
------------------------------------------------------------------------------

   ---------------------------------------------------------------------------
   Goldplating:
   ---------------------------------------------------------------------------
   - look into new channels for use in the defaults.

   ---------------------------------------------------------------------------
   New filter_items subroutine:
   ---------------------------------------------------------------------------
   - CHANNEL LINKS: Check if the links in a channel's rss are relative
     ("/relative/click.html") and turn them into $channel->{htmlurl} .
     "/relative/click.html" (absolute versions). We're gonna move this
     to a "filter_feeds" routine for the templates, so this bugfix is
     deferred.
   - EXTERNAL LINKS: There are still some feeds that our A HREF regexp
     doesn't seem to catch for fixing the TARGET of the link. Add about
     a hundred or so feeds, and you'll inevitably find one. Why? NOTE:
     Is related to "'d HREFs. We're gonna move this to a "filter_feeds"
     routine for the templates, so this bugfix is deferred.
   - HASHED TITLE AND DESCRIPTIONS: check for HASH again.
   - SHOW ONLY X ITEMS FROM CHANNEL? No HOWTO would be needed.

   ---------------------------------------------------------------------------
   Known bugs:
   ---------------------------------------------------------------------------
   - we don't check for errors on whether the mirror/filesave
     actually works, causing fun nonesucheries on permission errs, etc.
     i bet we could make the mirror code more readable in the first place.
   - fix up the logging messages for downloading. they're confusing.
   - figure out what to do with expat?

   ---------------------------------------------------------------------------
   Crappy code that needs to be fixed/I'm not satisfied with:
   ---------------------------------------------------------------------------
   - I don't like how AmphetaDesk::Channels::load_channel will accept
     a filename or a string - that seems kludgely to me. at most, we should
     accept a filename or a URL. support for strings was added to handle
     AmphetaDesk::Channels::add_url, but that needs redoing too. all of
     this would be rewritten/revamped when we did our AmphetaDesk::Parser.
   - the add_url routine is a mess, going all over the place. it really
     should be rewritten to handle HTML autodiscovery, subscription
     regardless of parsability, etc. the current code also doesn't support
     multiple subscriptions where one of the URLs is https://.
   - I don't like the AmphetaDesk::MyChannels::load_my_channels code,
     especially the odd kludge to check for raw data or a filepath.
     I'm not sure this is really something I can fix without totally
     overhauling the import/add_url code (which will happen anyways).
   - I don't really like the default subscription lists being in
     MyChannels.pm anymore. Or the fact that our dates are bogus
     defaults. Should we move these back to an external file and
     use our "import" to bring them in as the user's list?
   - do the save/load_my_channels/settings really need to accept
     $passed_path? there's really no time the defaults aren't used.
   - the writing of the OPML file in XML::Simple sucks, since
     a value has to be "" not undef, else XML::Simple tries to set
     the undef as a new hash element. this should get better if
     we can write our own printer when we switch to Ampheta::Parser.

   NOTE: We're gonna have to rebuild the OS X gui for the next version of
         AmphetaDesk, because we're using a new version of expat to fix the
         UTF8-BOM segfault. We don't need a new binary for Win32, since we
         "fix" the UTF8-BOM bug by cheating in our load_channel code.


------------------------------------------------------------------------------
SOMETIME SOON (no specific date or release. just nice to get done).
------------------------------------------------------------------------------

 - BACKUPS: We do no backups of our important files (myChannels.opml,
   mySettings.xml, and the log file). We should add a directory beneath
   /data/ called 'backups' and create weekly backups there (4 weeks worth?).
   I've got code for this already (check the nhpr backup scripts you wrote).

 - CGI SUPPORT: $os would become Internet, $connection would be STDOUT?
   Lots of people want this. The biggest addition would be security settings
   and user accounts. How do you handle $HOME directories?

 - CHANNEL DBMS: In future versions of AmphetaDesk, our channel information
   should be stored in DBM files. This will supersede anything we do with
   channel meta items (and the matching XML). The primary benefit (above
   and beyond the channel meta items, which allow "new items in a feed",
   "show only items newer than Y days or hide items older than X" days)
   is that we'll be able to keep a local store of RSS data, allowing people
   to create aggregate feeds on keyword, search through months worth of
   data, and so forth.

 - CHANNEL DUPLICATES: Check for duplicate code fails on query strings
   and 'www' or no 'www'. Query strings we can't really take care of.

 - CHANNEL META ITEMS: This is all in Morbus' head, but has been discussed
   on the mailing lists, in #disobey, and was partially implemented in
   deus' AmphetaOutlines. It requires Digest::MD5. ([e-mail],
   [e-mail], [e-mail], [e-mail],
   [e-mail], [e-mail], [e-mail],
   [e-mail] [e-mail], [e-mail],
   [e-mail], [e-mail] [e-mail],
   [e-mail], [e-mail], [e-mail]
   [e-mail])

 - CHANNEL GROUPING / ORDERING: There needs to be some way of ordering
   the channels. The current plan is to allow channels to be grouped
   under different headings like "Tech News", "Health News" and so on.
   Categories can be sorted differently than the channels within. See
   http://sourceforge.net/mailarchive/message.php?msg_id=1726994.
   ([e-mail], [e-mail], [e-mail], [e-mail],
   [e-mail], [e-mail], [e-mail])

 - CHANNEL HTML: Strip all HTML? Allow only these tags? Problem tags
   include title, script, and textarea. Tables are relatively dangerous
   as well, but should be remotely restricted to their own channel.
   Tags to allow: a h* strong em i blockquote ol ul li img p br?

 - CHANNEL IGNORING: Some people would like to remember channels that
   have piqued their interest, but don't want to actually see their
   items in the channel listing until they call for it (they'd still
   like to stay subscribed). The idea would be to have the title of the
   channel shown at the end of the listing, with the number of new
   items ("Wired News - 7 new items since last viewed"). People would
   be able to click on the title to see the items that have been
   hidden from view. This would be possible with categories.
   ([e-mail])

 - CHANNEL LISTS: A user should be able to specify which language they
   are interested in, and get a list of feeds back to them. This will
   be pretty easy once the Syndic8 export is happening, although
   we'll probably have to move to the OCS exports instead (which,
   in the long run, should give us more power anyways).
   ([e-mail])

 - CHANNEL LISTS: Reenact downloading of updated channel lists. This
   will require a script on disobey.com that will suck down the data
   and gzip, and then in AmphetaDesk, it'll have to Compress::Zlib to
   extract the data to the right location.

 - CHANNEL LISTS: Reenact "are you subscribed?" check for the lists.
   Not entirely sure what the visual stimulus should be, but a check
   box probably isn't it (since that'd cause a 'duplicate channel'
   error upon submittal). Perhaps the absence of a checkbox and
   something that says "you're currently subscribed, click here
   to unsubscribe"?

 - CHANNEL LISTS: Should be searchable. Standard AND/OR/NOT and URL
   specific searches should be possible. Can we just hook into the
   Syndic8 API? Not so sure I like the co-dependance factor. The
   sad thing is, this isn't a perfect solution since the data
   available from the feeds aren't adequately described, making
   bad search results. Whatever we do, we'll have to include some
   sort of warning about bad data. ([e-mail],
   [e-mail], [e-mail], [e-mail])

 - CHANNEL LISTS: Sorting needs to be smarter, by not alphabetizing by
   "A Channel Name" or "The Channel Name" and instead just "Channel Name".
   ([e-mail])

 - CHANNEL SORTING: On the main channels page, a user should be
   able to sort alphabetically, by date downloaded or added, etc.
   This is pretty easy due to our get_my_sorted_channels routine.
   We'll have to include a default setting under My Settings.

 - CHANNEL UPDATES: Force update of all channels. This needs to be GUI
   capable for Windows, and also accessible from the web pages. Should
   we add the "Refresh Channel List" to the web pages as well? How will
   we stop people from force-refreshing multiple times? Perhaps only
   allow force refreshes of one channel at a time? Perhaps we implement
   this as a query string like ?update_channels ([e-mail]).

 - CGI: Not entirely sure how we're going to handle this, but
   CGI.pm is anxious to convert URL parameters into parameters
   for our own usage. This starts causing problems when people
   are trying to ?add_url, and the URL in question has CGI
   parameters as well. As of right now, we're telling people
   to decimal encode those parameters, but not everyone can.

 - COMMAND LINE FLAGS: -nobrowser, -port, -quiet?

 - CONTENT-TYPES: We send out text/html for a content-type on
   any non-image request to our webserver. This should get smarter
   and default to text/plain for everything but /?html?/.

 - CUSTOMIZABLE EXPORTS: The collected data of all the downloaded
   RSS files should be exportable to a template based file, or to
   a webserver, or even to email. This all needs to be customizable,
   as well as having a pretty happy GUI. ([e-mail],
   [e-mail])

 - DOCS: add faq? "how to find your proxy server", "common zip problems".

 - DOCS: convert the various AmphetaModules into POD so that they can
   be converted into HTML documentation for the webpages. easier for
   me to refer developers to them.

 - DOCS: finish template documentation (add data available for use
   within the templates, how to use unknown data from RSS feeds,
   and so on and so forth).

 - DOCS: finish up the rest of the new v0.93 developer documentation.

 - DOCS: get deus to write a bit about the "to create a mac os x binary"
   on the build page (how are icons built? can he blurb a bit about
   the shell files, and how those are run?)

 - DOCS: write up HOWTOs for "only xxx letters", "show last x channels",
   "local weather", "using LWP::Simple to get data from other sites", etc.

 - ETAGS: From http headers,  these are unique id's that change only
   when the content has changed. Here's an example (the collection
   of these tags would live in the channel DBM file). While we're adding
   this, we should throw in Conditional Get support as well (reevaluate
   our HEAD, too). http://fishbowl.pastiche.org/archives/001132.html

 - EXPAT: Currently, we strip all BOM from UTF-8 feeds in load_channel.
   This is because some versions of expat can crash if fed one of these
   feeds. This isn't much of a problem under Windows or MacOS/OSX, since
   we control the version of expat shipped. However, we need to detect
   what version of expat is installed on Linux systems, and if we can't,
   strip BOM stuff.

 - GOLDPLATING: Our note() and error() routines should join() the
   entire @_ so that we don't have to use concat tricks to send
   out long error messages. Probably gigamiliseconds of savings.

 - GUI (Mac): Remove the MacPerl menu's. Stick with our own.

 - GUI (Mac): open_url should hook into MacPerl::Internet (see Fry email).

 - GUI (Win): Allow an option to shut off the GUI for Windows?

 - GUI (Win): Minimize on start, minimize on X? ([e-mail])

 - GUI (Win): Good points. I'll certainly add the "last time checked"
   sort of thing to the Systray, but the "display the log file" is a bit
   stickier. I may just run a shell sort of command ("open
   /data/AmphetaDesk.log") and have Windows open the program (or the "Open
   With..." window) that is defaulted to handle .log extensions. Also,
   change tray icon to reflect downloading status? ([e-mail])

 - HTTP AUTHENTICATION: Allow auth to protected Yahoo! Groups and
   generically, any site that requires a username and password.
   ([e-mail], [e-mail], [e-mail])

 - INSTALLATION: This is primarily important for Linux users and
   whenever we do CGI based installs, but the wrapper script should
   show what modules are not installed, and give some instructions
   on how to download them (either for the system, or into our
   AmphetaDesk lib/ directory). IOW, we shouldn't fatally die
   impolitely anymore, but offer users a chance to reconcile.
   We also need to check for version numbers - we use Vars for
   CGI.pm, which is relatively new enough that some don't have
   the correct module version.

 - LOGGING: We should use ISO dates and times on our output,
   especially since the logfiles now stick around for 250k.

 - MULTITHREADED EVERYTHING: The webserver should be multi-threaded,
   the download code should be multi-threaded, and if we could get the
   GUI loaded in it's own process, that'd be insanely great. Whether this
   is all magically possible on every system has yet to be tested. More
   than likely, this feature won't be possible until we can see if
   threads work under Classic Mac support, since it doesn't support
   forks(). I really think this is a cross-platform pipedream.

 - MULTI-USERS: Make AmphetaDesk support multiple users with one
   install. There's a whole crapload of email on this. ([e-mail],
   see also SF feature request)

 - MY CHANNELS OPML UPDATES: If we receive a 301 HTTP Redirect, we should
   update the URL in the myChannels.opml (we've got initial code from
   Klaus to do this).

 - OPEN URLS: In some systems, due to bad networking or crackwhore
   firewalls, people can't use 127.0.0.1 or localhost to access
   their AmphetaDesk. We should examine ways of detecting the user's
   IP address on the box (if they have one), and give them the option
   of using that IP instead (we've instructed people how to hard code
   their IP address in the past).

 - PLUGIN APIs. Already planned, and there's a quickie email here on it:
   http://sourceforge.net/mailarchive/forum.php?thread_id=827854&forum_id=9314
   ([e-mail], [e-mail])

 - PROXY AUTODISCOVERY: IE registry settings? .pac files? (Autodiscovery
   of .pac files is documented somewhere.  The idea being that a client
   can make a DNS request for a particularly named file from a specially
   named host.  Searching on .pac should be helpful - Kearney).

 - RSS PARSING: We really should revamp our RSS parsing system. The current
   plan is to create AmphetaDesk::Parser which would discover new parsers
   at AmphetaDesk::Parser::RSSv090.pm, ::RSSv200.pm, and so on. This would
   allow people to drag and drop new parsers, and AmphetaDesk would understand
   them at startup. In addition to this, each parser will also return a list
   of errors concerning why the feed is incorrect (even though we'll be
   ultraliberal in what we accept, we'll now give the user a chance to
   report the error). Autodiscovery of parsers should be possible by
   browsing through %INC for AmphetaDesk::Parser's location, and then
   getting a readdir of parsers sitting in AmphetaDesk/Parser. When
   we implement this, our OPML.pm should take over the processing
   done by AmphetaDesk::ChannelsList::load_channels_by_letter.

 - RSS PARSING: Namespace support. We'll have to write a
   "get_prefix_from_url" namespace routine which will take a passed
   URL (like the NS URL for mod_admin or any of the other RSS modules),
   and then return the actual prefix used. This will allow our templates
   to easily print/display that information.

 - RSS PARSING: Support RDFS descriptions for modules? We'd have to keep
   a local store of them shipped with AmphetaDesk, but we'll always let
   those be overridden with data we find online.

 - SECURITY: There needs to be access lists for the internal webserver -
   to either allow X, Y, Z ip/dns. We've got the "localhost or not"
   code, but need some more flexibility.

 - STDOUT: Check over all the notes and errors and responses. Make sure
   they're doing what they should be and are well written. Probably a
   good bet to extract them all and compare them side by side. I'm being
   really anal about this since it's vitally important to communicate.

 - TEMPLATES: Look into BROKEN under Text::Template which we can use
   to display custom errors when there's a template error. This makes
   a much better output than the blanket Perl error currently.

 - TEMPLATES: We need to add a front end to downloading and using templates.
   These templates need to come with info files, so that people can make
   custom settings, version numbers, and who to contact for more information.

 - TEMPLATES: Possible ideas for templates: a left-handed bar and click as
   per Mozilla or IE sidebars. Clicking on the left would open in the main
   window on the right. A collapsible DHTML interface like shown here:
   http://www.webreference.com/dhtml/column12/. ([e-mail]) One
   guy ([e-mail]) also had us load into IE's search bar:
   >javascript:Q='';if(top.frames.length==0)Q=document.selection.createRange
   >().text;void(_search=open('http://127.0.0.1:8888/index.html','_search'))
   Also, look into deus_ex's outliner version and implement. I've got
   a Lynx template from Ransom Smith - include that one. See if we
   can rip off a three-pane NNWL with frames. And finally:
   http://www.kottke.org/plus/misc/images/netnewswire.gif

 - TEMPLATES: Figure out why whitespace is needed after our [$ and
   $] delimiters for Text::Template. This worked fine on simple
   test cases, but something in our setup causes a necessity for
   something to be on the delimiter line, even if just a comment.

 - USAGE DATA: Perhaps allow anonymous upload of AmphetaDesk data that
   would find out which feeds are most popularly subscribed too, and
   yadda yadda. This would kinda be a DayPop sorta thing, but more
   on feeds themselves as opposed to actual items. This lends great
   credence to an Amazon like recommendation system. ([e-mail])

 - USAGE LEARNING / KEYWORDS: A number of people want AmphetaDesk to
   get smarter. They want to say "These are things I'm interested in",
   and have AmphetaDesk list those items at the top of the screen, or
   in a feed all by itself (for website export). The keyword idea would
   be simple to use - perhaps we could add a button next to each item
   like "add to keywords". AmphetaDesk would then parse that item, adding
   uncommon words to the user's keyword list. In some respects, this could
   be sent to a central server, and AmphetaDesk could recommend new
   channels that match their interests (although, at this point, this
   is kinda satanic. Perhaps we could include the relevant data into the
   massive channel list ourselves?). ([e-mail])

 - VERSION CHECKING: Should be an XML file, not plain text. This will all
   be handled once we jump into Compress::Zlib and start looking into
   streaming updates. Streaming updates are a long long way off, but at
   least moving to XML will allow us greater flexibility. We should also
   move data/internal/version.txt to just plain old data/version.txt,
   getting rid of the internal/ directory entirely.

------------------------------------------------------------------------------
KNOWN RARE/MINOR BUGS (nothing major, but should be fixed).
------------------------------------------------------------------------------

 - MAC AND BROWSER CHOOSING: Mac users can't choose a browser besides
   the default. Alan has sent a MacEvent solution, we should try and
   integrate that. [This has never been brought up so is low priority.]
 - MAC OS X AND MISSING IMAGE: "Just one question: there's a large empty
   area at the top of the AmphetaDesk OS X window (above the status line
   that you can 'dink' open to see the log)." ([e-mail];
   screenshot available. ask morbus). Using 10.1.5 and the April Dev Tools.
   Confirmed using Jaguar as well.